ARFA  ORDER  NO.  189-1 


1*4 


^kEMORANDUM 


M-5016-ARPA 

CjOCTOBER  1966 


REAL-TIME  RECOGNITION  OF 
HANDPRINTED  TEXT 


CLEARINGHOUSE 
FOR  FEDERAL  SCIENTIFIC  AND 
TECHNICAL  INFORMATION 


Hardcopy 

iSov 


Microfiche 

*  .50 


5X 

pp 


Gabriel  F.  Groner 


/'D 


PREPARED  FOR: 

ADVANCED  RESEARCH  PROJECTS  AGENCY 


r 

liu 


NOV  4  j36S 


bmid. 


W  V  -  -  V .< 

CL. 


%  K"H  I  I  \J 

SANTA  MONICA  •  CALIFORNIA- 


BEST 

AVAILABLE  COPY 


MISSING  PAGE 
NUMBERS  ARE  BLANK 
AND  WERE  NOT 
FILMED 


ARPA  ORDER  NO.  189-1 


MEMORANDUM 

RM-5016-ARPA 

OCTOBER  1960 


REAL-TIMF  RECOGNITION  OF 
HANDPRINTED  TEXT 

Gabriel  F.  Groner 


This  research  is  supported  by  the  Advanced  Research  Projects  Agency  under  Contract 
No.  SD-79.  Any  views  or  conclusions  contained  in  this  Memorandum  should  not  be 
interpreted  as  representing  the  official  opinion  or  policy  of  ARPA. 

1  )1STR I BUTI ON  STATEM ENT 

Distribution  of  this  document  is  unlimited. 


Rii  in) 


(700  MAIN  s:  •  SANt  A  MONICA  •  CAUI>0*NIA  •  «C«0« 


iii- 


PREFACE 


This  Memorandum  details  a  scheme  for  the.  computer 
recognition  of  handprinted  text.  This  scheme  allows  an 
on-line  computer  user  to  print  text  naturally,  and  have 
it  recognized  accurately  in  real-time.  The  investigation 
employed  the  RAND  tablet.  This  study  is  but  one  facet 
of  an  overall  search  for  techniques  i o  increase  the  fa¬ 
cility  of  the  man-computer  interface. 
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SUMMARY 


This  Memorandum  describes  a  scheme  which  permits  an 
on-line  computer  user  to  print  text  naturally,  and  have 
it  recognized  accurately.  This  scheme,  which  recognizes 
53  symbols,  enables  a  user  to  communicate  data  and 
directives  to  a  computer  by  using  only  a  RAND  tablet 
with  its  pen  as  an  input  device. 

The  tablet  provides  high  resolution  point-by-point  pen 
location  data  which  the  recognition  scheme  analyzes  as  a 
symbol  is  being  drawn.  This  analysis  describes  a  hand¬ 
printed  symbol  primarily  as  a  time  sequence  of  features. 
Identification  is  based  on  a  data-dependent  sequence  of 
tests.  Symbols  are  separated  from  one  another  according 
to  their  identifications  and  relative  positions. 

The  scheme  is  evaluated  by  determining  how  accurately 
it  recognizes  isolated  symbols,  i.  d  how  much  time  users 
need  to  compose  computer  code  on-line. 
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I .  INTRODUCTION 


During  Che  past  ten  years,  there  has  been  consider¬ 
able  interest  in  the  automatic  recognition  of  handwritten 
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and  machine-printed  symbols.  Most  of  these  studies 

have  involved  computer  recognition  of  an  already  completed 

symbol;  a  few  have  involved  special  electronics  for  the 
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analysis  and  recognition  of  characters  or  words  as 
they  are  being  written. 

Only  recently,  with  the  advent  of  on-line  computer 

systems,  have  computers  been  used  for  the  real-time 

recognition  of  handprinted  symbols. ^  ^  from 

13 

Bernstein’s  recent  work,  real-time  schemes  to  date  have 
been  concerned  only  with  the  recognition  of  single, 
isolated  symbols.  None  of  the  real-time  schemes  (to  the 
author's  knowledge)  have  been  used  in  a  problem  solving 
environment,  or  have  utilized  contextual  information  for 
recognition. 

This  Memorandum  describes  a  symbol  recognition  scheme 
which  allows  an  on-line  computer  user  to  print  text 
naturally,  and  have  it  recognized  accurately,  as  he  prints 
it.  The  scheme  responds  very  quickly  even  though  it 
recognizes  a  fairly  large  set  of  symbols.  Moreover,  it 
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imposes  few  constraints  on  sty;  ,  speed,  or  position  of 
writing.  It  makes  use  of  contextual  information  to  dis¬ 
tinguish  symbols  which  cannot  be  distinguished  by  shape 
alone.  This  scheme  has  been  used  daily  in  an  experimental 
system  at  The  RAND  Corporation  for  writing  computer  code, 
editing,  and  developing  flowcharting  symbology.  The  symbol 
recognition  scheme  is  written  in  IBM  System/360  Assembly 
Language  and  runs  on  an  IBM  System/ 360  Model  40. 

The  symbol  set  consists  of  the  upper-case  Latin 
alphabet,  the  numbers,  the  symbols  +,  -,  =,  /,  (,  ),  [,  ], 

*,  $,  .,  ,,  ’  ,  A,  >,  <,  and  a  scrubbing  action  which  causes 
erasure  of  underlying  symbols.  The  scheme  also  recognizes 
a  set  of  flowcharting  symbols,  bu**  these  will  not  be  dis¬ 
cussed.  When  any  other  symbol  is  drawn,  it  is  either 
identified  as  one  of  the  above,  or  as  a  "cannot  interpret." 
This  symbol  set  is  sufficiently  large  to  enable  a  user  to 
communicate  data  and  directives  to  a  computer  by  using 
only  a  pen-like  instrument. 

The  recognition  scheme  is  designed  so  that  the  user 
can  concentrate  on  his  problem,  rather  than  on  extraneous 
operational  mechanics.  It  therefore  accommodates  a  variety 
of  printing  styles,  allows  for  the  printing  of  several 
symbols  in  quick  succession,  and  responds  quickly.  Finally, 
any  errors  made  by  the  scheme  may  be  corrected  easily. 
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II.  USER  INTERACTION 


A  user  communicates  with  the  computer  via  a  RAND 
14 

tablet  in  conjunction  with  a  cathode  ray  tube  (CRT)  dis¬ 
play.  The  tablet  hardware  consists  of  a  horizontal  10-in. 
square  writing  surface  and  a  pen- like  writing  instrument. 
Figure  1  depicts  a  user  interacting  with  this  hardware. 

As  the  user  moves  the  pen  near  the  tablet  surface,  a  dot 
on  the  CRT  follows  the  pen  motion--this  direct  feedback 
helps  him  position  the  pen  for  pointing  or  drawing.  When 
he  presses  the  pen  against  the  tablet  writing  surface,  a 
switch  in  the  pen  closes,  thereby  notifying  the  central 
processing  unit  (CPU)  of  a  "pen-down"  action.  As  he  moves 
the  pen  across  the  tablet,  the  pen's  track  is  displayed  on 
the  CRT--the  pen  "point"  thus  seems  to  have  "ink."  When 
the  pen  is  lifted,  tne  pen  switch  is  opened,  thereby  notify¬ 
ing  the  CPU  of  a  "pen-up"  action,  and  "inking"  ceases.  A 
new  user  easily  adjusts  to  watching  the  pen  motion  on  the 
CRT  while  actually  moving  the  pen  off-screen  on  the  tablet. 
Furthermore,  he  finds  it  convenient  to  have  no  part  of  the 
CRT  "working  surface"  covered  by  his  hand  or  pen. 

The  user  must  print  plainly;  i.e. ,  not  use  curls  or 
scrolls,  and  not  drag  the  pen  between  strokes.  The  letter 
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0  must  be  written  with  a  slash  to  distinguish  it  from 
the  number  0,  the  letter  I  with  serifs  to  distinguish  it 
from  1,  and  the  letter  Z  with  a  crossbar  to  distinguish 
it  from  2  (these  conventions  are  commonly  followed  when 
writing  on  coding  forms) .  The  user  may  otherwise  print 
in  his  normal  style.  When  the  user's  "inked"  symbol  is 
recognised,  it  is  replaced  by  a  hardware-generated  version 
He  may  change  this  symbol  at  any  time,  merely  by  writing 
over  it. 

Although  the  user  may  write  anywhere  on  the  tablet 
suiface,  he  must  follow  some  conventions  of  size  and 
position  just  as  he  would  when  writing  on  a  piece  of  paper 
The  display  format  subtly  conditions  him  by  displaying  a 
recognized  symbol  in  a  standard  size  (full-size  symbols 
are  about  3/ 16- in.  high)  at  the  standard  location  (one 
of  a  possible  3370)  nearest  to  the  center  of  his  "inked" 
version  of  the  symbol.  Since  the  user  writes  next  to 
standard-size  symbols  uniformly  spaced  along  lines,  he 
soon  learns  to  adjust  his  printing  accordingly.  This 
symbol  size  and  separation  are  about  the  same  as  those 
commonly  required  for  printing  on  coding  forms. 
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III.  ANALYSIS  OF  THE  DATA 

The  purpose  of  data  analysis  is  to  extract  those 
features  most  valuable  for  discrimination  among  the 
allowable  characters;  i.e.,  those  features  which  have 
consistent  values  ''ver  variations  of  a  particular  symbol, 
yet  which  have  different  values  f :r  different  symbols. 
Since  a  single  scheme  recognizes  the  printing  of  many 
users,  any  of  whom  may  print  sloppily,  consistency  is 
assured  only  if  a  symbol  can  be  described  by  a  few  gross 
features . 

The  on-line  nature  of  this  recognition  scheme  enables 
processing  of  the  data  point-by-point  as  the  pen  is  moved 
across  the  writing  surface.  In  order  to  minimize  time 
and  sto  age  requirements,  therefore,  the  scheme  extracts 
features  as  the  data  arrive. 

THE  DATA 

Pressing  the  pen  against  the  writing  surface  acti¬ 
vates  the  symbol  recognition  scheme  by  indicating  the 
start  of  a  stroke.  As  the  pen  is  moved  across  the  writing 
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surface,  the  recognition  scheme  is  notified  of  its  posi¬ 
tion  every  4  msec.  Finally,  when  the  pen  is  lifted  off 
the  surface,  the  recognition  scheme  is  notified  that  the 
stroke  is  completed.  Each  pen  position  is  accurate  to 
about  0.005  in.  The  hardware  thus  provides  a  very  de- 
tailed--abc.-ut  100  data-points--time-seqaential  descrip¬ 
tion  of  each  stroke. 

SMOOTHING  AND  THINNING 

The  scheme  smooths  the  data  by  averaging  a  newly 
arrived  data-point  with  the  previously  smoothed  data- 
point,  thus  reducing  the  noise  due  to  the  discreteness 
of  the  pen  location  as  measured  by  the  tablet.  Smooth¬ 
ing  is  based  on  the  equations 

xsi  -  !xsi-i +  ki 

and 


3  1 

Yc .  =  tY0 .  .  +  7Yd  . 
Si  4  Sl-1  4  Ri 
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where 


^  Vi 

X_.,  YD .  =  coordinates  of  the  i  raw  data-point 
Kl  Rl 


and 


til 

X„.,  Y„ .  =  coordinates  of  the  i  smoothed  data-point. 

Si  Si  r 

Thinning  is  the  process  of  removing  some  of  the  data- 
points  from  the  pen  track.  This  is  accomplished  by  comparing 
the  position  of  a  new  smoothed  data-point  with  the  position 
of  the  last  point  in  a  thinned  track.  If  these  points  are 
sufficiently  far  apart,  the  analysis  scheme  accepts  the 
smoothed  point  as  part  of  the  thinned  track;  otherwise,  it 
is  discarded.  Thinning  eliminates  small  perturbations  in  the 
track,  and  reduces  the  data  processing  requirements  by 
drastically  reducing  (by  a  factor  of  seven  or  so)  the  number 
of  data-points.  Thinning  is  described  by 
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XSi  ’  YTj  YSi  * 


if 


s  A  , 


and/ or 


*  A  , 


where 


are  the  coordinates  of  the 
data-point 


.  th 
J 


thinned 


and 


A  is  a  parameter  of  the  analysis  routine. 

The  recognition  scheme,  which  expects  symbols  to  be  drawn 
i.bo.’t  3/16- in.  high,  uses  A  =  0.02  rn  is  value  of  A  is 
large  enough  to  eliminate  insignificant  uata-points,  yet 
small  enough  to  maintain  the  essential  characteristics  of 


the  track. 
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Figure  2  is  a  photograph  of  a  display  generated  by 
a  program  which  did  the  following: 

1)  Replotted  without  any  processing  each  sampled 
point  of  a  figure  drawn  at  the  tablet; 

2)  Magnified  the  original  figure  eight  times; 

3)  Smoothed  the  track; 

4)  Thinned  the  track  with  A  =  0.01  in. 

THE  CURVATURE  FEATURE 

Curvature  is  the  most  obvious  track  characteristic 

which  is  independent  of  position  and  size,  and  yet  which 

15 

describes  the  track's  shape.  Freeman  has  suggested  that 
a  useful  approximation  to  curvature  is  the  sequence  of 
quantized  directional  segments  generated  by  the  points  in 
a  thinned  track.  Kuhl^  and  Bernstein^  have  used  this 
approximation  in  their  character  recognition  schemes. 
Bernstein,  in  fact,  found  it  unnecessary  to  use  the  dura¬ 
tion  of  each  quantized  direction  but,  rather,  simply 
listed  changes  in  quantized  direction.  Whereas  Kuhl  and 
Bernstein  both  used  eight  possible  directions,  the  recog¬ 
nition  scheme  described  here  uses  only  four.  Four  direc¬ 
tions,  used  in  conjunction  with  other  features,  provide 
sufficient  description  for  recognition,  yet  result  in 
fewer  symbol  variations  than  do  eight  directions. 
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V 


Inches 

(a)  as  drawn 

(b)  magnified 

(c)  magnified  and  smoothed 

(d)  magnified,  smoothed,  and  thinned 


FIG.  2  -  Display  of  a  Stroke 
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When  a  new  point  (the  j  )  is  accepted  in  the  thinned 
track,  a  quantized  direction  is  computed  using  these  in¬ 
equalities  : 

L>  lf  IKtj  -  k  |YTj  -  YTj-l'  > 

a)  direction  is  right  if  X^  -  X^  ^  ^  0  , 

b)  direction  is  left  if  X^,.  -  X^,.  ^  <  0  ; 


2)  if 


l*Tj  '  Vl' 


Y 


Tj-1 


a) 

b) 


direction  is 
direction  is 


up  if  -  YTj-1  *  0  , 


down  if  Y^  -  Y^^  <  0  . 


If  the  same  *.  action  occurs  twice  in  succession,  and  is 
not  the  same  as  the  last  direction  listed  in  the  sequence, 
then  it  is  added  to  the  list;  otherwise  it  is  discarded. 
This  requirement  causes  further  thinning. 

Small  hysteresis  zones  (about  16°  wide)  around  the 
quantized  direction  zone  borders  prevent  the  quantized 
directions  in  the  approximate  track  description  from 
switching  back  and  forth--say  from  right,  to  up,  to  right-- 
when  part  of  a  track  is  drawn  along  one  of  these  borders . 
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If,  for  example,  the  pen  is  moving  to  the  right,  it  mast 
next  proceed  at  an  angle  steeper  than  53°  in  order  for  the 
direction  to  change  from  right  to  up. 

Applying  these  rules  to  the  thinned  track  of  Fig.  2 
results  in  the  direction  sequence  left-down-right-down- 
left-up  . 

INKING  AF)  CORNER  DETECTION 

The  CRT  displays  the  pen  track  as  "ink"  constructed 
of  a  sequence  of  directional  segments  originating  at  the 
pen-down  location.  Sixteen  possible  directions  are  re¬ 
quired  to  provide  the  user  with  sufficient  "ink"  detail. 

A  segment  is  added  to  the  "ink"  each  time  a  new  thinned 
data-point  arrives. 

This  same  sequence  of  directions  is  used  to  detect 
sharp  corners .  A  corner  is  detected  whenever  the  pen 
moves  in  the  sane  (±1)  16-direction  for  at  least  two 
segments,  changes  direction  by  at  least  90°,  and  then 
proceeds  along  the  new  direction  (±1)  for  at  least  two 
segments.  The  change  in  direction  must  take  place  either 
immediately  or  through  a  one- segment  turn.  Applying  this 
test  to  the  track  in  Fig.  2  detects  corners  in  the  upper- 
left  and  lowermost  parts  of  the  stroke. 
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SI2.E  AND  POSITION  FEATURES 

As  a  stroke  is  drawn,  its  x  (horizontal)  and  y 
(vertical)  extremes  are  continuously  updated.  When  the 
pen  is  lifted,  therebv  indicating  the  completion  of  the 
stroke,  the  analysis  scheme  uses  these  extremes  to  calcu¬ 
late  the  symbol's  height  and  width  in  fractions-of-an-inch, 
its  aspect  ratio  (ratio  of  height  to  width) ,  and  its  center 
relative  to  the  tablet  origin.  It  divides  the  rectangular 
area  defined  by  the  symbol  extremes  into  a  4  x  4  grid.  The 
starting  (pen-down)  and  ending  (pen-up)  points,  as  well  as 
the  corner  locations,  are  then  each  encoded  as  lying  in  one 
of  these  16  areas,  thereby  locating  them  relative  to  the 
symbol.  Figure  3  shows  the  4x4  grid  defined  by  the  track 
in  Fig.  2.  This  track  has  the  following  description: 

1)  Six  4-direction  segments  (left-down-right-down- 
left-up)  encoded  as  2-3-0-3-2-1; 

2)  Corners  at  positions  7  and  15; 

3)  Starting  point  at  position  0; 

4)  Ending  point  at  position  0; 

5)  Height  =  0.30  in.; 

6)  Width  =  0.22  in. ; 

7)  Aspect  rati.o  -  1.36; 

8)  Center  at  x  =  3.15  in.,  y  =  1.75  in. 


t 

i 


* 
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COMPUTING  REQUIREMENTS 

The  delta  analysis  described  above  takes  up  about  40 
percent  of  the  available  computing  time  in  the  limiting 
case,  where  each  data-point  becomes  a  member  of  the 
thinned  track.  Smoothing  and  thinning  requires  about  20 
percent  of  this  analysis  time,  inking  30  percent,  and 
corner  detection  16  percent.  The  remainder  of  analysis 
time  is  for  computing  quantized  directions,  updating  x  and 
y  extremes,  and  bookkeeping.  The  smoothing,  thinning,  and 
inking  functions  could  be  handled  by  hardware  or  by  an 
I/O  processor.  Corner  detecting  could  be  deferred  until 
stroke  completion  without  requiring  further  storage  (since 
it  is  based  on  the  ink  description ,  which  is  stored  in  .ny 
case),  and  without  noticeably  increasing  the  time  between 
stroke  completion  and  recognition.  With  these  changes, 
a  single  user  would  require  only  about  15  percent  of  an 
IBM  System/360  Model  40. 
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IV.  DECISION-MAKING 

Several  characteristics  of  the  decision-making 
scheme  should  be  pointed  out  before  it  is  described  in 
detail.  When  a  stroke  is  completed,  its  description  is 
added  to  those  of  the  other  strokes  belonging  to  the  same 
symbol.  The  decision-making  scheme  identifies  the  partial 
symbol  corresponding  to  this  set  of  strokes,  but  does  not 
display  its  identification  at  this  time.  Such  a  partial 
symbol  is  considered  complete  when  the  user  pauses,  draws 
a  somewhat  distant  stroke,  or  draws  a  stroke  which  cannot 
be  combined  with  the  partial  symbol  to  form  one  of  the  53 
allowable  symbols.  The  symbol  is  then  displayed.  The 
decision-making  scheme  thereby  separates  symbols  from  one 
another  and  recognizes  them  as  they  are  written. 

The  identification  of  a  symbol  is  based  on  a  data- 
dependent  sequence  of  tests.  At  each  step  in  the  decision¬ 
making  process  there  are  several  potential  identifications. 
Some  of  these  are  eliminated  by  testing  cne  of  the  features 
of  the  track.  The  particular  test  applied  at  any  step 
depends  on  the  set  of  possible  identifications  at  that 
step,  and  on  those  characteristics  of  the  track  which  have 
already  been  examined.  The  decision-making  scheme  thus 
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has  a  tree  structure.  Its  original  design  was  based  on 
an  examination  of  the  handwriting  of  four  users.  The 
author  changes  ts  structure,  to  accommodate  additional 
symbol  variations,  as  he  acquires  more  experience. 

IDENTIFICATION  OF  SINGLE  STROKES 

The  first  test  groups  the  single  stroke  partial 

symbols  according  to  shape.  This  is  accomplished  by 

locating  a  sequence  of  direction  segments  in  a  table. 

Only  a  few  segments  should  be  looked  up  in  the  table 

because  each  additional  segment  increases  the  table  length 

by  a  factor  of  four  (recall  that  there  are  four  quantized. 

directions).  However,  if  too  few  are  looked  up,  the  test 

will  not  sufficiently  reduce  the  number  of  possible  stroke 

identifications  in  each  shape  group.  A  good  compromise 

is  to  look  up  the  first  four  direction  segments  of  a  track. 

If  a  track  has  fewer  than  four  direction  segments,  it  is 

encoded  such  that  the  last  segment  is  repeated  until  there 

are  a  total  of  four.  The  table  is  therefore  256  entries 

2  3 

long--160  (4  +  4*3  +  4‘3  +  4*3  )  of  which  correspond  to 
allowable  encodings.  For  this  test,  the  track  of  Fig.  2 
is  encoded  as  2-3-0-3  (left-down-right-down).  The  possible 
stroke  identifications  corresponding  to  this  table  entry 
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are  S,5,8,9,  and  "cannot  interpret."  Further  tests,  based 
on  this  particular  set  of  possibilities,  result  in  a  single 
identification . 

The  sequence  of  the  first  four  direction  segments  of 
a  track  provides  good  first-level  discrimination.  In  the 
present  table,  forty-five  of  the  160  possible  sequence 
encodings  result  in  immediate  identifications,  with  nc 
further  testing  necessary.  Another  50  sequences  result  in 
an  immediate  "cannot  interpret." 

To  allow  for  variations  in  handwriting,  each  allowable 
symbol  has  many  table  entries.  A  track  for  the  symbol  3-- 
wbich  has  one  of  the  most  complex  and  varied  shapes--may 
be  drawn  having  any  one  of  26  different  sequences  of  the 
first  four  direction  segments.  Figure  4  shows  some  of 
the  tracks  identified  as  the  symbol  3. 

Assuming  that  each  of  the  single  stroke  partial 
symbols  is  equally  likely,  and  that  each  four-direction 
segment  description  of  any  particular  symbol  is  equally 
likely,  the  probability  of  having  a  number  of  symbols  with 
the  same  table  exit  can  be  calculated.  Figure  5  shows 
this  probability  of  requiring  further  testing  to  dis¬ 
criminate  among  a  number  of  symbols  upon  exiting  from  the 
currently  used  table.  Approximately  30  percent  of  the 
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3  3  53 
^  *>? 


FIG.  4  -  Some  Tracks  Recognized  as  the  Symbol  "3". 
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direction  sequences  require  no  further  testing.  The 
probability  of  requiring  further  testing  to  discriminate 
among  as  many  as  four,  five,  or  six  symbols  is  fairly  high, 
because  several  symbols  might  have  the  same  first  four 
directions  in  common,  but  other  features  which  are  dif¬ 
ferent  . 

After  this  first  test,  the  stroke  is  either  recognized 
or  is  known  to  be  one  of  a  particular  set  of  two-to-seven 
symbols.  The  second  test  is  one  which  best  (in  the 
author's  judgment)  discriminates  among  these  particular 
symbols.  Depending  on  the  symbols,  this  test  distinguishes 
differences  in: 

o  The  directions  following  the  first  four  (e.g.,  to 
distinguish  some  ^s  from  some  3s); 

o  The  number  of  corners  (e.g.,  S  vs  ^) ; 

o  The  positions  of  the  starting  or  ending  points 
relative  to  the  symbol  extremes  (e.g.,  0  vs  6) ; 

o  The  aspect  ratio  (e.g.,  C  vs  £  )  ; 

o  The  size  (e.g.,  ^  vs  ±  ) ; 

o  The  position  relative  to  a  line  of  text 
(e.g.  ,  a  vs  )  . 

Testing  continues  until  the  number  of  possible  stroke 
identifications  is  reduced  to  one.  Figure  6  shows  the 
sequence  of  testing  required  for  the  recognition  of  the 
track  in  Fig.  2.  It  is  identified  as  the  symbol  8. 
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Table  Exit  Corresponding  to 
Direction  Sequence  =  2-  3  -  0-  3 


FIG.  6  -  Recognition  of  the  Track  of  Figure  2. 
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The  number  of  tests  required  for  recognition  is  an 
interesting  parameter  because  it  measures  the  time  and 
computer  memory  required  for  decision- asking .  The  proba¬ 
bility  of  requiring  some  number  of  tests  for  recognition 
can  be  calculated  if  it  is  assumed  that  each  allowable 
single  stroke  partial  symbol  is  equally  likely  and  that 
each  decision  tree-exit  corresponding  to  a  particular 
symbol  is  equally  likely.  Figure  7  shows  the  probability 
of  requirir.  further  testing  after  performing  a  number 
of  tests  in  the  existing  decision  tree.  In  more  than 
half  of  all  stroke  recognition  sequences,  only  two  tests 
are  required  to  arrive  at  a  stroke  identification.  Figure 
7  shows  that  ambiguity  among  several  possible  stroke 
identifications  is  always  resolved  after  six  tests. 

RECOGNITION  OF  MULTIPLE- STROKE  SYMBOLS 

The  recognition  of  a  multiple-stroke  symbol  is  based 
on  the  identifications  and  relative  positions  of  its  con¬ 
stituent  strokes,  but  not  on  the  or d  i.r  in  which  they  are 
drawn.  This  is  accomplished  by  crc ss-li nking  the  tree 
structure.  Each  symbol  has  several  descriptions,  and 
therefore  may  be  drawn  in  several  ways.  For  example,  a 
set  of  three  horizontal  strokes  and  one  vertical  stroke 
is  recognized  as  the  symbol  E  regardless  cf  the  writing 


n,  the  Number  of  Tests  Already  Performed 


FIG.  7  -  The  PrcLviLility  of  Requiring  Further  Testing  After  u  Tests, 
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order.  The  symbol  E  may  also  be  written  as  an  "L-like" 
stroke  and  two  horizontal  strokes,  a  "P-like"  stroke 
and  two  horizontal  strokes,  a  "[-like"  stroke  and  one 
horizontal  stroke,  or  as  a  single  "£  -like"  stroke. 

In  many  situations,  each  stroke  requires  only  a 
general,  rather  than  a  precise,  identification.  For  ex¬ 
ample,  if  a  stroke  known  to  be  part  of  a  multiple-stroke 
symbol  is  drawn  vertically  downward  (±45°) ,  it  need  only 
be  identified  as  a  vertical  stroke,  rather  than  as  a 
1,  ),  (,  or  /.  This  reduces  the  compounding  of  errors 
that  would  otherwise  result  from  incorrect  identifica¬ 
tions  of  single  strokes,  and  also  simplifies  tie  decision¬ 
making. 

Just  as  the  starting  and  ending  points  of  a  single¬ 
stroke  symbol  are  encoded  according  to  their  positions 
relative  to  the  x  (horizontal)  and  y  (vertical)  extremes 
of  that  stroke,  the  starting  and  ending  points  of  each 
stroke  in  a  multiple-stroke  symbol  are  encoded  according 
to  their  positions  relative  to  the  x  and  y  extremes  of 
the  whole  symbol.  These  encoded  positions  differentiate 
those  symbols  which  are  comprised  of  the  same  combination 
of  strokes.  At  most,  four  symbols  have  the  same  set  of 


strokes . 


A  symbol  comprised  of  two  strokes  identified  as  ver¬ 
ticals  is  a  K,  V,  X,  or  Y.  This  symbol  is  recognized  as 
a  K  if  one  of  the  strokes  has  both  its  starting  point  and 
ending  point  in  the  leftmost  quarter  of  the  symbol.  It 
is  recognized  as  a  V  if  both  of  its  ending  points  are  in 
the  middle  part  of  the  lower  qua:  er  of  the  symbol.  It 
is  w  Y  if  it  has  one  ending  point  which  is  neither  in  the 
leftmost  quarter,  ncr  in  the  lower  quarter.  Otherwise, 
this  symbol  is  an  X.  Other  ambiguities  among  multiple- 
stroke  symbols  are  similarly  resolved. 

SEGMENTATION  INTO  SYMBOLS 

As  the  user  writes  a  line  of  text,  the  recognition 
scheme  decides  when  a  partial  symbol  is  completed,  recog¬ 
nizes  and  displays  it,  and  begins  the  analysis  of  the 
next  symbol  as  it  is  being  written.  The  scheme  separates 
the  set  of  strokes  making  up  a  completed  partial  symbol 
from  the  first  stroke  in  the  next  symbol  by  considering 
timing,  and  the  geometric  extents  and  identifications  of 
the  partial  symbol  and  the  following  stroke. 

If  a  prespecified  time  elapses  following  the  end  of 
the  most  recent  stroke,  the  corresponding  partial  symbol 
is  considered  completed  regardless  of  what  follows.  This 
between- symbol  time  delay  must  be  greater  than  the  maximum 
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expected  time  delay  between  two  strokes  belonging  to  the 
same  symbol--0.3  sec  has  proven  sufficient  for  experienced 
users.  A  sufficient  pause  between  symbols  eliminates  the 
need  to  test  for  geometrical  separation.  This  procedure 
reduces  normal  writing  speeds  by  a  factor  of  about  one- 
half,  but  results  in  the  most  accurate  segmentation. 

A  partial  symbol  is  considered  completed,  regardless 
of  timing  or  position,  if  it  cannot  be  combined  with  the 
following  stroke  to  form  an  allowable  symbol.  The  symbols 
8,  Q,  A,  and  E  are  examples  of  partial  symbols  which  can¬ 
not  be  combined  with  any  other  stroke  to  form  an  allowable 
symbol.  If  a  partial  symbol  can  be  combined  with  some 
strokes,  but  not  with  others,  then  the  following  stroke 
is  potentially  identified  on  the  basis  of  its  first  four 
quantized  direction  segments.  A  test  is  then  made  to  de¬ 
termine  if  the  partial  symbol  can  be  combi.neu  with  a  stroke 
having  this  particular  potential  identification  to  form  an 
allowable  symbol.  A  partial  symbol  identified  as  a  T,  for 
example,  may  be  combined  with  a  vertical  stroke  to  become 
an  A,  H,  K,  or  *,  or  may  be  combined  with  a  horizontal 
stroke  tc  become  an  F  or  I,  but  cannot  be  combined  with  a 
potential  C,  U,  or  2.  In  some  situations,  the  following 
stroke  must  be  identified  more  precisely.  A  circular 
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stroke  followed  by  a  normal  size  "/-like"  stroke,  for 
example,  may  be  identified  as  the  letter  0;  but  a  circular 
stroke  followed  by  a  short  "/-like,;  stroke  will  be  iden¬ 
tified  as  the  number  0  followed  by  a  comma. 

Strokes  which  are  written  in  quick  succession,  and 
which  can  be  combined  to  form  an  allov  able  symbol  are 
tested  for  geometric  separation.  Two  such  schemes  for 
separating  symbols  geometrically  have  been  investigated 
at  RAND.  One  of  these  assumes  that  the  writing  surface 
is  an  85  x  42  array  of  symbol  spaces,  and  that  each  symbol 
is  drawn  in  a  different  space.  The  other  scheme  assumes 
that  the  strokes  in  a  symbol  overlap  (or,  in  some  cases, 
are  close  to  one  another) ,  but  do  not  overlap  with  strokes 
belonging  to  another  symbol. 

Fixed  Symbol  Spaces 

This  scheme  tests  to  see  if  the  centers  of  a  partial 
symbol  and  that  of  the  next  stroke  lie  in  the  same  symbol 
space.  If  they  do  not,  it  separates  them  unless  the 
partial  symbol  is  a  single  vertical  stroke  lying  just  to 
the  left  of  the  following  stroke.  If  they  do  lie  in  the 
same  space,  it  considers  them  as  part  of  the  same  symbol 
unless  the  stroke  is  a  vertical  in  the  rightmost  quarter 
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of  the  symbol  space,  and  the  partial  symbol  cannot  be 
combined  with  a  r ighthand  vertical  to  form  an  allowable 
symbol.  These  adjustments  on  the  basic  test  allow  for 
some  misplacement  of  vertical  strokes 

The  user  must  be  nade  aware  of  the  symbol  space  posi¬ 
tions  so  that  he  may  space  his  printing  accordingly.  This 
is  accomplished  by  displaying  a  series  of  dimly  ] it  vertical 
bars,  each  at  a  border  of  a  symbol  space. 

The  primary  disadvantage  of  this  scheme  is  that  small 
displacements  of  strokes  relative  to  the  symbol  spaces 
cause  segmentation  errors.  Such  displacements  may  occur 
when  the  user  writes  quickly,  or  when  the  hardware  dis¬ 
places  the  "ink"  slightly,  thereby  misinforming  the  user 
as  to  where  he  is  writing.  A  further  disadvantage  is  that 
the  vertical  bars  are  distracting. 

Relative  Symbol  Positions 

This  scheme  tests  for  the  overlapping  or  adjacency 
of  the  partial  symbol  and  the  following  stroke.  If  the 
horizontal-extent,  of  the  stroke  lies  within  the  horizontal- 
extent  of  the  partial  symbol,  it  considers  them  parts  of 
the  same  symbol.  Partial-overlap  causes  the  partial  sym¬ 
bol  and  the  following  stroke  to  belong  to  the  same  symbol, 
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unless  their  centers  are  farther  apart  than  a  maximum 
allowable  distance.  No-overlap  separates  the  partial 
symbol  and  the  following  stroke,  unless  one  (or  both)  is 
a  single  vertical  stroke  and  their  centers  are  not  sep¬ 
arated  by  more  than  a  maximum  allowable  distance.  The 
allowable  separation  distances  are  based  on  the  normally 
expected  symbol  size. 

The  maximum-distance-between-centers  test  enables 
the  separation  of  two  symbols,  such  as  T/  or  R2.  which 
happen  to  overlap  because  they  are  written  close  together. 
When  symbols  having  a  vertical  stroke  are  written,  the 
vertical  frequently  does  not  overlap  the  remainder  of 
the  symbol.  The  position  test  in  the  no-overlap  case 
takes  this  into  consideration.  It  is  particularly  use¬ 
ful  when  |  |  will  be  made  an  H  or  N  by  the  next  stroke, 
and  also  when  two  parts  of  the  same  symbol--e.g. ,  the  1 
and  3  of  B--are  just  slightly  disconnected. 

The  disadvantages  of  this  scheme  are  that  it  may 
separate  two  parts  of  the  same  symbol  which  are  not  vrritten 
close  enough,  and  that  it  may  i.jt  separate  two  different 
symbols  which  are  written  such  that  they  overlap  slightly. 
Its  advantage  is  that  it  works  about  as  well  as  the  fixed- 
symbol-space  scheme,  yet  does  not  require  the  display  of 


symbol  space  separators.  It  uses  the  displayed  symbols 
to  cue  the  user  about  how  large  to  write  and  about  how 
far  apart  to  place  symbols.  Since  the  symbols  are  dis¬ 
played  with  fixed  spacing,  users  tend  to  inadvertently 
imbed  bl^nk  spaces  unless  they  are  provided  with  some 
strong  indication  of  the  location  of  the  symbol  spaces. 
This  scheme  therefore  works  best  when  the  vertical  bars 


are  displayed. 
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V .  EVALUATION 


One  measure  of  the  value  of  the  recognition  scheme 
is  the  accuracy  with  which  it  recognizes  individual  symbols. 
Since  the  goal  of  the  scheme  is  to  facilitate  highly  inter¬ 
active  man-computer  communication,  a  more  direct  measure 
of  its  value  is  the  time  required  to  perform  some  writing 
task  on-line. 

Three  groups  of  users  participated  in  the  following 
two  experiments.  The  first  group  consisted  of  six  users 
who  have  had  considerable  experience  with  the  symbol 
recognition  scheme.  The  design  of  the  decision-making 
scheme  was,  in  fact,  based  on  the  handwriting  of  four  of 
these  users.  The  second  group  consisted  of  seven  engineers 
and  programmers  who  have  had  experience  using  the  RAND 
tablet,  but  have  not  previously  written  with  the  symbol 
recognition  scheme.  The  third  group  was  made  up  of  nine 
engineers,  programmers,  and  secretaries  who  have  had  no 
previous  experience  with  either  the  tablet  or  the  recog¬ 
nition  scheme. 

ACCURACY  OF  RECOGNITION 

Each  user  in  the  second  and  third  groups  participated 
in  a  half-hour  training  session  prior  to  the  testing  phase 
of  this  experiment.  The  session  began  with  an  explanation 
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of  how  to  use  the  tablet,  what:  symbols  are  recognized, 
the  use  of  the  scrub,  etc.  The  user  was  then  allowed  to 
write  whatever  he  wanted  for  twenty  minutes--most  users 
printed  words,  phrases,  or  the  alphabet,  usually  repeat¬ 
ing  a  symbol  until  it  was  recognized  correctly.  During 
the  next  ten  minutes,  the  user  attempted  to  print  each 
of  the  symbols  while  the  author  pointed  out  '-.ow  he  might 
achieve  more  success.  The  users  in  the  first  group  did 
not  receive  any  special  training,  since  they  were  already 
familiar  with  the  tablet  and  recognition  scheme. 

Since  this  experiment  was  concerned  with  isolated 
symbols,  the  decision-making  scheme  was  adjusted  (during 
the  testing  phase)  to  separate  symbols  on  the  basis  of 
time  only.  Each  user  was  instructed  to  carefully  print 
the  list  of  symbols  five  times,  waiting  for  each  symbol 
to  be  recognized  before  printing  another.  (The  apostrophe 
was  not  included  because  the  users  were  given  no  position 
cues,  and  therefore  could  not  be  expected  to  write  an 
apostrophe  different  from  comma.)  Each  user  thus  printed 
each  of  52  symbols  five  tines;  53  responses  were  possible 
(including  "cannot  interpret").  All  errors  (including 
user  errors  such  as  writing  0  rather  than  0  for  the  letter 
0)  were  recorded. 
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The  average  recognition  rate  for  the  group  with  ex¬ 
perience  in  using  the  symbol  recognition  scheme  was  93 
percent  correct.  The  lowest  "score"  in  this  group  was 
90  percent;  the  highest,  9C  percent.  The  average  "score" 
among  those  experienced  previously  with  only  the  RAND 
tablet  was  88  percent,  with  a  low  "score"  of  82  percent 
and  a  high  "score"  of  92  percent.  The  average  "score" 
among  those  with  no  previous  experience  whatsoever  was 
87  percent,  with  a  low  of  81  percent  and  a  high  of  93 
percent.  The  recognition  rates  were  not  only  high,  but 
quite  uniform.  In  fact,  the  "scores"  for  users  in  the 
second  and  third  groups  are  indistinguishable.  The  time 
it  takes  to  learn  to  use  *  . aware — with  its  separate 

tablet  writing  surface  and  CRT  working  surface-- therefore 
appears  to  be  less  than  one-half  hour.  The  users  in  the 
first  group  probably  scored  somewhat  higher  than  the  others 
because  they  were  more  familiar  with  the  scheme,  and  be¬ 
cause  it  was  designed  to  accommodate  their  writing  styles. 

Many  of  the  errors  were  due  to  the  misrecognition  of 
(,  ),  [,  and  ]  --  (,  for  example,  was  frequently  confused 
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wich  1,  C,  or  L.  The  asterisk  was  also  frequently  missed 
because  its  shape  is  not  well-defined,  and  it  therefore 
has  many  variations.  Some  users,  not  familiar  with  coding 
form  conventions  found  it  difficult  to  learn  to  write  the 
letter  0  with  a  slash  to  distinguish  it  from  the  number  0, 
yet  learned  to  print  Z  with  a  crossbar,  and  I  with  upper 
and  lower  bars.  The  remainder  of  the  errors  were  due  to 
confusions  between  7  and  >,  0  and  C,  5  and  S,  +  and  T,  and 
a  wide  variety  of  other  pairs  of  symbols. 

EASE  OF  OPERATION 

Since  this  recognition  scheme  i?  designed  for  use  in 
a  problem-solving  environment,  an  experiment  was  performed 
in  which  programmers  used  the  scheme  while  solving  simple 
coding  problems.  A  user  was  given  flowcharts  (Fig.  8), 
and  was  asked  to  write  the  corresponding  computer  code  for 
one  problem  directly  on-line  (using  only  the  recognition 
scheme)  and  that  for  the  other  off-line  (using  pencil  and 
paper) .  The  times  required  for  solution--a  measure  of  the 
usefulness  of  the  recognition  scheme  versus  pencil  and 
paper--were  recorded. 

Since  time  was  important,  this  experiment  used  the 
decision-making  scheme  which  separates  symbols  according 
Lo  identity  and  relative  position  as  well  as  time.  Users 


FIG.  8  (a)  -  Flowchart  of  Problem  A  Coded  In  the  Experiment:  A  Procedu 
for  Computing  Social  Security  Tax. 
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FIG.  8  (b)  -  Flowchart  of  Problem  B  Coded  In  the  Experiment: 

A  Method  of  Sorting  Three  Numbers  Into  a  Descending 
Sequence  of  Their  Magnitudes. 
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could  therefore  print  quickly  without  pausing  between 
symbols.  Vertical  bars  were  displayed  to  indicate  symbol 
spacing.  Users  were  allowed  to  make  use  of  all  editing 
facilities-- i . e . ,  change  a  symbol  by  writing  over  it; 
erase  a  symbol,  space,  or  line  with  a  scrubbing  action; 
use  the  symbol  a  to  insert  symbols  between  others  already 
printed  on  a  line;  and  use  the  symbol  >  to  obtain  a  blank 
line  between  two  printed  lines  of  text. 

Eight  programmers  participated  in  this  experiment. 
Four  from  the  first  group  (described  above)  coded  the 
problems  in  IBM  System/ 360  Assembly  Language.  Each  prob¬ 
lem  required  the  printing  of  approximately  two  hundred 
symbols  in  this  language.  Two  programmers  each  from  the 
second  and  third  groups  coded  the  problems  in  FORTRAN- - 
this  required  the  printing  of  about  one-hundred-fifty 
symbols  for  each  problem.  Half  of  the  users  in  each  group 
coded  Problem  A  directly  onto  a  coding  form  and  Problem  B 
directly  on-line,  using  the  tablet  hardware  and  recog¬ 
nition  scheme;  the  others  coded  Problem  B  onto  a  coding 
form  ^nd  Problem  A  on-line. 

Table  I  summarizes  the  results  of  this  experiment. 
Each  entry  is  an  average  for  two  users.  The  inexperienced 
(second  and  third  group)  users  spent  a  large  part  of  their 
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Table  I 

TIMES  TO  CODE  THE  PROBLEMS  OF  FIG.  8 


1 — 

Time  ( 

min) 

Group 

Problem 

Coding  Form 

On-line 

A 

8  3/4 

12 

± 

E 

9 

9  1/2 

2  &  3 

3 

7 

12  1/2 

B 

4 

9 

on-line  problem  solving  time  trying  to  communicate  with 
the  recognition  scheme.  The  experienced  users,  who  had 
harder  problems  (assembly  language  versus  FORTRAN)  to 
solve,  were  not,  however,  slowed  down  very  much  by  -\e 
recognition  scheme. 

mhe  subjective  findings  of  this  experiment  are,  per¬ 
haps,  more  important  than  the  numerical  results.  The 
accuracy  of  recognition  here  was  much  lower  than  in  i_he 
previous  experiment,  partly  because  the  scheme  had  to 
separate  symbols  geometrically;  but  also  because  the  in¬ 
experienced  users  had  forgotten  some  of  what  th<  v  had 
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learned  about  the  scheme,  and  because  all  of  the  users 
were  less  careful.  All  users  found  it  difficult  to  print 
symbols  as  small  and  as  closely  packed  as  the  hardware 
generated  them--85  across  a  10- in.  width.  They  tended 
to  print  oversize  symbols,  thus  causing  the  scheme  to 
introduce  blank  spaces  or  to  recognize  a  single  multiple- 
stroke  symbol  as  two  symbols. 

The  editing  facilities  helped  to  compensate  for  the 
difficulties  with  symbol  recognition.  The  users  found  it 
convenient  to  change  a  symbol  by  writing  directly  over  it, 
or  to  erase  an  entire  line  with  a  single  scrubbing  action-- 
editing  procedures  which  are  much  more  difficult  to  do 
with  pencil  and  paper  than  with  the  tablet  hardware.  The 
symbol  A  was  frequently  used  to  insert  one  or  more  symbols 
between  two  others.  Although  the  symbol  >  was  used  on 
only  two  occasions  to  •>'nsert  a  new  line  between  two  others, 
this  editing  feature  is  potentially  valuable  for  composing 
solutions  to  more  difficult  problems. 

Thus,  this  on-line  recognition  scheme  with  editing 
facilities  appears  to  be  a  useful  problem-solving  aid, 
particularly  as  the  users  become  more  experienced,  and 
the  problems  become  more  difficult.  In  a  problem-solving 
situation,  the  editing  facilities  give  the  user  much  greater 


flexibility  than  pencil  and  paper.  The  recognition 
scheme  is  natural  to  use,  provides  constant  feedback, 
and  makes  so  few  errors  that  it  is  not  bothersome. 


-43- 


VI.  DISCUSSION 


This  recognit  con  scheme  meets  its  primary  objective 
of  enabling  any  user  to  communicate  naturally  with  a  com¬ 
puter.  A  user  is  not  distracted  by  any  operational 
mechanics  bit,  rather,  may  concentrate  on  his  problem. 

All  communications  are  made  with  a  single  device--a  pen¬ 
like  instrument.  A  user  may  write  anywhere  on  a  hor\- 
zontal  writing  surface.  The  recognition  scheme  has  been 
extended  to  recognize  such  flowcharting  symbols  as 
rectangles,  circles,  triangles,  and  diamonds.  Thus,  the 
user  may  draw  free- form  flowcharts  as  well  as  text,  using 
only  a  pen. 

The  recognition  program  responds  quickly  and  is 
efficient  in  storage.  When  the  time-delay  normally  used 
to  separate  symbols  is  set  to  zero,  the  lifting  of  the 
pen  and  the  display  of  a  recognized  symbol  are  apparently 
simultaneous.  The  recognition  program--including  the 
data  analysis  and  decision-making  routines,  and  data 
storage;  but  net  display  or  editing  rout ines- -requires 
about  twenty- four  hundred  32-bit  words  of  memory. 

The  major  shortcoming  of  the  present  scheme  is  its 
difficulty  in  recognizing  quickly  written  text:  it  has 
difficulty  in  separating  overlapping  symbols  and  in 
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combining  disconnected  parts  of  a  single  oymbol.  Further¬ 
more,  since  quickly  written  symbols  tend  to  be  distorted, 
they  are  misrecognized  more  often  than  carefully  drawn 
symbols.  These  problems  are  due,  in  part,  to  the  small 
size  and  close  packing  of  the  symbols.  They  can  be 
relieved  by  allowing  larger  between-symbol  spaces. 

Another  difficulty  with  the  recognition  scheme  is 
that  only  a  person  familiar  with  the  computer  program  can 
add  a  new  symbol  or  new  symbol  description.  Such  changes 
are  complicated  because  they  frequently  require  several 
coding  changes  in  the  cross-linked  tree  structure  of  the 
decision-making  scheme.  A  useful  variation  of  this  scheme 
would  therefore  be  based  on  a  data-dependent  sequence  of 
tests,  but  would  permit  automatic  changes. 

Finally,  the  advantages  and  disadvantages  of  pen- 
location-by-pen-  location  feature  extraction  should  be 
clarified.  This  procedure  is  useful  for  real-time  recog¬ 
nition  because  it  minimizes  the  time  delay  between  symbol 
completion  and  identification,  yet  produces  a  valuable 
set  of  features.  It  may,  however,  result  in  symbol  vari¬ 
ations  not  introduced  by  a  scheme  which  extracts  features 
after  a  symbol  is  completed.  Such  variations  arise  be¬ 
cause  some  symbols  may  be  drawn  either  clockwise  or 
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counterclockwise,  portions  of  some  symbols  may  or  may  not 
be  retraced,  some  symbols  may  be  constructed  of  various 
numbers  and  sequences  of  strokes,  etc.  Perhaps  some  other 
set  of  dynamically  extracted  features  would  prove  as  use¬ 
ful  for  discrimination  as  the  present  set  without  intro¬ 
ducing  as  many  variations. 
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